Skip to content

fix(analytics): measure open and click rates against tracked emails - #534

Open
joanproductivo wants to merge 4 commits into
useplunk:nextfrom
joanproductivo:fix/533-tracked-rates
Open

joanproductivo wants to merge 4 commits into
useplunk:nextfrom
joanproductivo:fix/533-tracked-rates

Conversation

@joanproductivo

Copy link
Copy Markdown
Contributor

Description

Bug

With Settings → Email tracking set to Marketing only or Disabled, the open and click rates on the dashboard and in Analytics still count emails that were sent without tracking. Those emails carry no open pixel and no rewritten links, so they can never record an open or a click, and they dilute the rate: one untracked transactional email plus one opened campaign email reads 50% instead of 100%. The dashboard also counted every email row created in the period as sent (queued, failed, inbound and mailbox-simulator mail), so it disagreed with Analytics even with tracking enabled.

Root cause

Whether an email is tracked is decided at send time from the project's mode and the email's stream (EmailService.shouldTrackEmail, plus the no-tracking configuration set toggle in SESService), but no stats query takes it into account: ActivityService.getStats divided opens by _count._all, and AnalyticsService.toMetrics divided them by sent.

Solution

  • EmailService.trackedSourceTypes(mode) derives the streams a mode tracks from shouldTrackEmail and the same toggle SESService applies, so stats and the send path follow one rule. EmailService.trackedStreams(projectId) resolves it for a project through the cached ProjectService.id, and falls back to "everything tracked" if the project cannot be read.
  • Analytics: a tracked count next to sent in every count, timeseries point and mailbox provider row. Open and click rates are measured against tracked, and opened/clicked are filtered with the same predicate, so neither rate can pass 100% after the mode changes. The tracked streams go into the SQL as literals: a bound parameter cast to "EmailSourceType" is cast again on every row, which made countsByStream about three times slower in my measurements.
  • Campaign leaderboard: a campaign's type sets the stream of every email it sends, so a campaign the current mode does not track gets tracked: 0; tracked campaigns keep matching their campaign page.
  • Dashboard: same scope as Analytics, with a groupBy on (sourceType, simulated) served from emails_projectId_createdAt_stats_idx. In a separate commit, "Emails sent" counts the outbound emails that went out; the onboarding banner, which keys off that number, now hides once the first email actually goes out rather than when it is queued.
  • The tracking mode is part of both cache keys, so changing it does not serve stats computed under the old one.
  • Web: each metric knows the volume it needs (sent, or tracked for open, click and click-to-open). With nothing tracked, the figure shows "—" instead of 0%: overview tiles add "Not tracked", the chart explains that none of the emails were sent with tracking, and the stream table, leaderboards and providers show "—".
  • Docs: the tracking guide explains how the rates are measured under each mode.

Projects with tracking Enabled get exactly the same Analytics numbers as before (there is a test for it). There is no schema change, migration or new index: every column the queries read is already in the INCLUDE of the stats index.

Worth knowing

  • Emails don't record whether they were tracked, so a period is classified with the project's current mode: changing the mode reclassifies the emails shown, including when switching back to Enabled. Stamping a tracked column at send time, the way simulated is stamped, would make this exact and would also cover per-email overrides like feat: per-template, per-campaign and per-send tracking overrides #433. I kept this PR migration-free; trackedSourceTypes would still be the fallback for rows sent before such a column. Happy to follow up with that if you prefer it.
  • Campaign pages (CampaignService.getStats) still measure against sentCount; I left them out of scope.
  • feat: capture email with project recipient suppression rules #522 also touches getStats and emailScope (a status filter), so one of the two will need a small rebase.

Type of Change

  • fix: Bug fix (PATCH version bump)

Testing

  • New tests: AnalyticsService.test.ts (8 cases), ActivityService.getStats (5), EmailService.trackedSourceTypes / trackedStreams (7), and metrics.test.ts for the web (3). On the current next the issue cases fail by assertion (25% or 50% instead of 100%), while the Enabled and mailbox-simulator cases already pass and keep passing.
  • Full suite locally against Postgres 16 and Redis 7 (Node 24): 1430 passed, against 1407 on next. Each of the four commits builds and passes its own tests. yarn lint reports no errors and yarn build --filter=api --filter=web succeeds. Prettier and ESLint are clean on the lines this PR changes.
  • Performance: 300,000 synthetic emails over three projects (100,000 of the measured project in range), the stats index built as background-index-builder does, and VACUUM ANALYZE. The analytics queries run through Prisma with real parameters (median of 15 runs): countsByStream 14 → 18 ms, daily timeseries 32 → 36 ms. The dashboard, as the SQL Prisma generates for the old aggregate and the new groupBy under EXPLAIN ANALYZE: 14 → 16 ms. All of them are an Index Only Scan on emails_projectId_createdAt_stats_idx with 0 heap fetches.

Checklist

  • PR title follows conventional commits format
  • Code builds successfully
  • Tests pass locally
  • Documentation updated (if needed)

Related Issues

Closes #533

Related: #531 (automatic pixel fetches counted as opens), #432 / #433 (per-email tracking override)

🤖 Generated with Claude Code

joanproductivo and others added 4 commits October 24, 2026 09:36
With email tracking set to "Marketing only" or "Disabled", transactional
emails (or all emails) go out without the open pixel or rewritten links,
yet the dashboard and analytics still divided opens and clicks by every
email sent, diluting both rates.

Open and click rates are now measured against the emails sent with
tracking under the project's current tracking mode. The streams a mode
tracks come from shouldTrackEmail, plus the no-tracking configuration set
toggle that SESService applies, so stats and the send path agree. Opens
and clicks are filtered the same way, so neither rate can pass 100% after
the mode changes. Everything stays within the columns of the stats index
and needs no schema change.

- analytics: a `tracked` count beside `sent` in every count, timeseries
  point and mailbox provider row; the campaign leaderboard tracks a
  campaign by its type, which sets the stream of all its emails
- dashboard: rates measured like analytics (tracked, not simulated)
- the tracking mode is part of both cache keys
- the chart measures opens and clicks against `tracked` as well

Projects tracking everything get the same analytics numbers as before.

Refs useplunk#533

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard counted every email row created in the period as sent:
emails still queued or that failed, inbound mail, and mail to the SES
mailbox simulator. "Emails sent" now counts outbound emails that went
out, the same scope as `sent` in analytics, so the dashboard and the
analytics page agree.

The onboarding banner keys off this number, so it now disappears once
the first email actually goes out rather than when it is queued.

Refs useplunk#533

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tracked

With nothing sent with tracking in a period or stream, open and click
rates read 0%, which says nobody opened rather than that nothing was
measured. Each metric now knows the volume it is measured on (`volume`:
emails sent, or emails sent with tracking for the figures built on opens
and clicks), and a figure with nothing to measure shows "—" instead:

- overview tiles show "—" with "Not tracked" under them, and the chart
  explains that none of the emails were sent with tracking
- the by-stream table, leaderboards and mailbox providers show "—"
- the dashboard's open rate card shows "—" and "Not tracked"
- deltas compare against the previous period's tracked volume

Refs useplunk#533

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explain how the dashboard and analytics measure open and click rates
under each tracking mode, that a period without tracked emails shows
"Not tracked", and that the current mode decides which emails count.

Refs useplunk#533

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard and Analytics open/click rates count untracked emails when Email tracking is "Marketing only"

1 participant